Long-branch attraction in species tree estimation: inconsistency of partitioned likelihood and topology-based summary methods

With advances in sequencing technologies, there are now massive am-ounts of genomic data from across all life, leading to the possibility that arobust Tree of Life can be constructed. However, “gene tree heterogeneity”,which is when different genomic regions can evolve differently, is a commonphenomenon in multi-locus datasets, and reduces the accuracy of standardmethods for species tree estimation that do not take this heterogeneity intoaccount. New methods have been developed for species tree estimation thatspecifically address gene tree heterogeneity, and that have been proven toconverge to the true species tree when the number of loci and number ofsites per locus both increase (i.e., the methods are said to be “statisticallyconsistent”). Yet, little is known about the biologically realistic conditionwhere the number of sites per locus is bounded. We show that when the se-quence length of each locus is bounded (by any arbitrarily chosen value), themost common approaches to species tree estimation that take heterogeneity ∗Department of Mathematics, University of Wisconsin–Madison, 480 Lincoln Dr, Madison WI53706†Department of Statistics, The University of Illinois at Urbana-Champaign, 725 S Wright St#101, Champaign IL 61820‡Department of Computer Science, The University of Illinois at Urbana-Champaign, 201 NorthGoodwin Avenue, Urbana IL 61801-2302 1arXiv:1803.02800v1[q-bio.PE]7Mar2018 into account (i.e., traditional fully partitioned concatenated maximum like-lihood and newer approaches, called summary methods, that estimate thespecies tree by combining gene trees) are not statistically consistent, evenwhen the heterogeneity is extremely constrained. The main challenge is thepresence of conditions such as long branch attraction that create biased treeestimation when the number of sites is restricted. Hence, our study uncoversa fundamental challenge to species tree estimation using both traditional andnew methods.